Enhancing educational technology practical course: Designing and validating tools for evaluating knowledge, performance, and satisfaction of public health students

Abstract Background The present study was conducted with the aim of designing and validating questionnaires for evaluating the public health students' knowledge, performance, and satisfaction (KPS) in the course of educational technology. Methods The present study was conducted with qualitative‐quantitative approach. Qualitative stage was done based on experts' opinion and related articles and questionnaires for developing, designing, and validating the tools. For validating the tools, face validity and content validity was done based on the opinion of the 10 expert panel. In quantitative stage, all initial validated tools were implemented on 50 students of public health for determining the reliability. The reliability of the tools was calculated using the internal consistency method of the items with Cronbach's ⍺ coefficient, and Kuder–Richardson. All analyses were conducted using SPSS‐19 software. Results In the field of qualitative research, an assessment tool consisting of 20 four‐option questions was designed. The content validity of this tool was confirmed based on the opinions of 10 educational experts, with CVI Total = 0.862 and CVR = 0.885. Additionally, the reliability of the tool was validated through testing on 50 public health students, resulting in a reliability coefficient of r = 0.780. Furthermore, for performance evaluation, four assessment tools each comprising 20 questions (totaling 80 questions) were designed. The validity values for these tools in the areas of PowerPoint (CVI = 0.981, CVR = 0.960), pamphlet (CVI = 0.866, CVR = 0.877), poster (CVI = 0.871, CVR = 0.906), role‐playing (CVI = 0.871, CVR = 0.980), and the reliability of the tools with Cronbach's ⍺ r > 0.80 were confirmed. In the satisfaction assessment section, a researcher‐made questionnaire consisting of 18 questions across seven domains was designed, with confirmed content validity (CVI = 0.949, CVR = 0.861) and reliability (r = 0.928). Conclusions It seems that this tool can be used to measure the KPS of students from the educational technology course of the health field and similar fields.


| INTRODUCTION
Universities, being a prominent academic institutes for investing in students as human capital, primarily focus on training and cultivating an effective workforce. 1Development of students in the health sector aims to improve service quality, necessitating the efficient training of students. 2One of the primary objectives in health and medical education should focus on instructing and preparing students in professional skills and personal growth, enhancing their selfdirected learning, and fostering social competencies. 3Ensuring the quality of students' performance in order to prevent the wastage of human and material capital and to have the ability to compete in the future world, where quality is the most important component for the survival of any organization, is an undeniable necessity. 4,5The most important goal of education in the fields of health professions is to develop and create qualifications, to train successful graduates who are capable of using what they have learned. 6Health and medical sciences encompass not only the imparting of theoretical knowledge but also the essential task of cultivating practical skills within the field.Public health, as a branch of medical sciences, is particularly instrumental in equipping students with a wide range of practical skills and competencies. 6Educational settings for practical training may not always provide ideal learning conditions.However, instructors have the opportunity to enhance the learning environment and maximize the benefits of education by creating favorable conditions, employing diverse strategies, and utilizing various teaching approaches. 7,8r this purpose, health science must integrate theoretical and practical knowledge of health based on logic derived from what is already known and what is gained from practical experience.This requires creating practical fields to promote the development of wise, critical, and thoughtful professionals. 9The practical learning process should enable students to develop the learned theoretical parts that integrate the application of knowledge with skills and attitudes in real-life practical situations. 10cordingly, one of the effective criteria of practical education is empowering students, followed by developing knowledge, attitude, and skills.These three components are of significant importance in complex learning and professional roles.Therefore, education can create necessary and needed competencies. 11,12Research indicates that designing integrated theoretical and practical activities based on real-world tasks leads to more effective education for students.[14] Among the fields of medical sciences is public health, where effective education in the dimensions of knowledge, attitude, and skills in these fields is of particular importance and should be paid attention to from the beginning of entering the university and in basic courses.
The educational technology course is one of the basic theoretical-practical subject that are offered to learn the practical skills of the students of this field, and it has a special place in the skills training of the learners and will play an important role in their job performance. 15ucational technology plays a crucial role in enhancing students' personality growth and motivation for learning. 16Studies 17,18 have highlighted the significance of innovative education in improving the readiness for learning.The application of educational technology, along with changes and innovative approaches, directly impacts the activities of both professors and students.Introducing innovations into the educational system through updates and changes is essential for the development of education.These innovations positively influence educational activities and contribute to overall improvement in the educational sector. 19e course of educational technology and educational aids, in terms of combining theory and practice, makes learning lasting and diversifying in the classroom.In fact, educational technology has played an effective role in improving the efficiency and quality of the teaching and learning process and the academic progress of public health students, 20 which increases the teaching and learning process of students by using media and educational aids. 21Various studies emphasize the positive effects of the potential power of educational technology on improving students' attitudes and performance. 22creased implementation of technology will increase students' comprehension of content and development of skills in such areas as analytical reasoning, problem solving, information evaluation, and creative thinking. 23One of the main approaches in health and medical education is the six-faceted SPICES model, which was presented by Harden with an analysis of the changing process and needs of modern health and medical sciences schools. 24The SPICES model in medical education focuses on key elements like studentcentered learning, problem-based learning, integrated or interprofessional teaching, community-based education, elective studies, and a systematic approach.This model aims to enhance teaching by promoting student-centered approaches, problem-solving skills, and interdisciplinary collaboration. 25One of the components of the SPICES model in medical education, particularly relevant in academic and field settings, is the emphasis on systematic training over opportunity-based training.This aspect highlights the importance of structured and methodical approaches to training, promoting integrative and problem-solving education within medical contexts. 26 the context of health education, it is essential to develop organized and purposeful tools to assess students' knowledge, performance, and satisfaction (KPS).In addition to measuring knowledge and performance, measuring student satisfaction is a crucial aspect of evaluating educational programs, including factors like faculty credentials, financial performance, learner outcomes, comparison with other programs, benchmarking, questionnaires, and hybrid models combining satisfaction and facilities assessment. 27edback from students plays a vital role in improving medical education programs, helping to identify areas for corrective measures and quality assurance. 28[31][32] To comprehensively evaluate students' KPS in medical education, it is crucial to utilize accurate and reliable tools that cover all dimensions of the course.Various methodologies and tools have been developed to measure student satisfaction in higher education, focusing on aspects like teaching quality, assessment methods, generic skills, learning experiences, and overall student contentment. 33,34 Asadabad College of Health Sciences, one of the practical courses along with the public health students' homework is the educational technology course.This course routinely starts with an overview of the theoretical material, and then the students learn from the department's experts, other peers, and self-study in the educational field.But there were no clear indicators to measure their effective learning criteria.This issue is especially important because the educational technology course is one of the courses, which is important in both theoretical and practical aspects for public health students.
Most of the studies conducted in relation to the validation of students' academic satisfaction scale by Turkzadeh et al. 35 the construction and validation of the student evaluation questionnaire of educational quality by Rezaei et al., the study of the validation of the Lent academic satisfaction scale in Iranian students by Setayeshi Azhari et al. 36 no reliable tool was found in this field.
Therefore, according to the importance of the performance and satisfaction of students and according to the experience of the researchers in presenting the practical part of the educational technology course and the students' opinions on how to teach and evaluate this unit; This study was conducted with the aim of designing and validating questionnaires of KPS of public health students in the course of educational technology.

| Study design
This research was carried out in two stages, qualitative and then quantitative.

| Qualitative stage
In the initial stage, the items of the questionnaires for measuring knowledge, satisfaction, and performance were designed using the review of texts, scientific and library resources, similar articles and interviews with three experts in the fields of medical education and health education and health promotion.In this study, 10 experts were selected to check face and content validity.Given that the questionnaires and checklists were researcher-made and tailored to the educational needs of the target group, experts in the areas of health education (3), medical education (3), and educational technology (4) were utilized to selecting the items and assess the content validity.Also, to determine the face validity of the initial questionnaire, the opinions of the students who had already completed this course were used.Also, to determine the face validity of the questionnaire, the opinions of the students who had already completed this course were used.

| Quantitative stage
In the quantitative stage to determine the reliability of KPS, three tools were implemented on all 50 public health students who had taken the course of educational technology in health.To determine reliability, the internal consistency of items with Cronbach's ⍺ coefficient and Kuder-Richardson's method was used.It should be noted that each semester, 25 students are enrolled in this course.To assess the questionnaire's reliability, we involved two cohorts who had completed the course in consecutive semesters.However, it is evident that this study needs replication and examination with a more extensive sample size.

| Sampling and participants
To determine the items of assessment tools (checklists and questionnaires) in the field of educational technology, a process was undertaken involving the identification of scientific sources and reputable references in the educational technology course.Subsequently, to establish the content validity of these tools, 10

| Educational content and duration
The duration of the educational technology course for public health students was one academic semester, and two consecutive courses for students in 2022 were selected to calculate the reliability of the tool.The educational technology course was presented in two parts: theoretical content presentation (short lecture) and the main practical part (individual and work group activities) in the field of designing and developing the PowerPoint, pamphlets, posters, and roleplaying (Table 1).

| Tools/Instruments
In this study, three different tools were used to measure knowledge, performance, and students satisfaction.The first tool was a 20-item multiple-choice questionnaire used to assess students' knowledge.The second tool consisted of four 20-item checklists to measure students' performance in practical activities, including producing Posters, Pamphlets, PowerPoints, and Role-playing.Overall, 118 items were used to measure theoretical and practical activities as well as student opinions in this study.

| Knowledge assessment questionnaire
This instrument included 20 four-choice questions in four areas; The principles of Nonreflective educational media, including (posters, pamphlets, Replica, etc. (seven questions), Reflective or fixed transparent educational media educational media (slides, APK, Overhead, PowerPoint, and so forth (seven questions), principles of the communication process to establish education (two questions), and the principles of role-playing (four questions)

N of items
Score range Total scoring scale
In the check of face validity, which is an initial indicator for checking content validity, the questionnaire was given to 10 experts and they were asked to express their opinions about the questionnaire and its external features (Table 1).At first, the content coverage of items was confirmed by researchers and teacher of the course based on the Learning Objective-content table and using the opinions of 10.Also, Milman's checklist was used in designing the Multiple-Choice Questions (MCQ).Then content validity was approved by 10 experts.

| Performance measurement questionnaire
In compiling the performance tool using reliable articles and sources and no with 0 score, and relatively or somewhat with 0.5 score.In total, the range of scores is between 0 and 40 (considering that the questions of each subscale were 20 questions and the score is between 0 and 1, so the score range is between 0 and 40), finally, the accepted cut-of-point was 20 (Table 1).

| Satisfaction assessment
This research-made questionnaire was prepared by studying the related articles in the field of research.This tool consists of 18 questions in seven  1).

| Face validity index
To determine the face validity of the questionnaires, a complete list of compiled items was given to 10 experts and including health education, medical education, educational technology; to comment on the questions.Also, after developing the initial version and before implementing it on the students, five public health students who had completed this course the previous year confirmed the face validity of the tools.

| Content validity
Content Validity Ratio (CVR): In determining the content validity index, 10 experts were asked to comment on the importance and necessity of all questionnaire items.In this approach, each item was examined based on the three-part spectrum: it is necessary, it is useful but it is not necessary, and it is not necessary.After collecting comments to evaluate the content validity index, CVR was calculated for each question according to the following formula; where Ne is the number of experts who consider an item necessary and N is the total number of experts.After the calculation, the result was compared with the entry criteria in the Lawshe table according to the number of experts (acceptable 0.62). 39

Content validity index (CVI):
The CVI is the degree of agreement between experts about the relevance, simplicity, and clarity of each item.That is, the number of experts who gave each item a score of 3 and 4 divided by the total number of experts, which is actually the ratio of agreement on the three mentioned indexes of each item.In this method, questions with a score higher than 0.79 are suitable, between 0.70 and 0.79 need to be modified, and less than 0.70 are unacceptable and should be removed. 40

CVI =
Number of raters giving a rating of 3 or 4 Total number of raters .
For knowledge assessment, the reliability of the questions was done using Kuder-Richardson's method.This formula is a special case of Cronbach's ⍺ test which is calculated for dichotomous scores.A high coefficient of KR-20, for example, more than 0.90 indicates a homogeneous test.The values of this formula can be from 0.00 to 1 (which is sometimes expressed as 0 to 100).The formula used in this method is given below.
To determine reliability of satisfaction and performance tools, the internal consistency method of questions with Cronbach's ⍺ was used.At the end, the data collected from 25 public health students were entered into SPSS.19 and calculated, Cronbach's ⍺ formula in this calculation was the following formula.The questionnaires were electronically prepared, and the links to the questionnaires were sent via email to the experts.
Due to the large number of questions and questionnaires, the tools were provided to experts gradually and with time intervals.

| Data analysis
For determining the CVI and CVR, data analysis is descriptive based on content validity formulas, and in the case of reliability, in the case of knowledge questions, using the Kuder-Richardson formulas, and in the case of multivalued questionnaires, the Likert scale is based on the internal consistency of items method and Cronbach's ⍺ coefficient was used.

| RESULTS
Based on the results of this study, questionnaires to measure students' KPS from the educational technology course with 118 questions and in three areas were designed and validated with the help of scientific sources and experts' opinions.

| Demographic characteristics
In the qualitative phase, the opinions of 10 experts specializing in medical education (3), health education (3), and educational technology (4) were used to measure content validity.
Six of them were female and four were male, and their teaching experience was between 3 and 21 years.
In quantitative stage after confirming the content validity of tools, to determine the reliability, these tools were implemented on 50 public health students who took the educational technology course.Of 50 participants 44 (88%) students were female and 6 (12%) were male.Their age range was between 19 and 22 years.Their grade point average (GPA) was between 14 and 18 and none of them had taken this course before or participated in educational technology workshops.

| Validation of knowledge assessment tools
Based on the results of the study, in the analysis of the CVI, the items that had CVR higher than 0.62, it was confirmed that the total CVR index was 0.885 and the total CVI was 0.862.Also, the value of CVI for relevance index (0.835),Clarity (0.863), and Simplicity (0.891) has been confirmed.The reliability of the questionnaire was 0.780 based on the Kuder-Richardson coefficient, which shows that all the knowledge questions (20 questions) have high reproducibility and were accepted (Table 2).

| Validation of performance assessment tools
Based on the study, a performance questionnaire with 80 questions was designed in four areas: poster, pamphlet, playing the role of a teacher in the training program and slides prepared in PowerPoint software.In the poster section, the CVR was 0.906 and the CVI index was 0.871 (Table 3).For pamphlet, CVR was 0.877 and the CVI index was 0.866 (Table 4).In the area of slides prepared in PowerPoint software, the total index of CVR was 0.960 and CVI was 0.981 (Table 5).Also in the part of role-playing as a teacher, the total CVR index was 0.980 and CVI was 0.871 (Table 6).In checking the reliability of the questionnaire; Cronbach's ⍺ coefficient was evaluated for each of the performance subscales included: poster and pamphlet tools (r = 0.880), PowerPoint (r = 0.871), and roleplaying (r = 0.890), which shows that this instrument has high reliability (Tables 3-6).

| Validation of satisfaction assessment tool
Based on the findings of the study; The questionnaire of students' satisfaction with the quality of the course was designed with 18 questions.The CVI was 0.949, the CVR was 0.861, and the overall reliability was 0.928, which indicates the high reliability of this tool and can be used to determine satisfaction with the quality of the practical course (Table 7).

| DISCUSSION
The purpose of the present study was to designing and validating questionnaires for evaluating the KPS of public health students in the practical course of educational technology.This study described the development and validation of tools based on face validity, content validity (CVI, CVR), and reliability (Internal consistency).
Knowledge: Based on the results of the study, the analysis of the CVI and CVR revealed important insights.Based on the Lawshe model if the experts are 10 members, items with a CVR higher than 0.62 confirmed. 39,41In this research a total CVR index of 0.885 which shows the very good validity of the knowledge measurement tool.
Also a total CVI of 0.862.Additionally, the CVI values for relevance (0.835), clarity (0.863), and simplicity (0.891) were confirmed, indicating strong content validity across these dimensions.In the examination of CVI, Waltz and Bausell stated that if the value of CVI is greater than 0.79, it is considered very good and acceptable.For values between 0.70 and 0.79, it is necessary for the items to be reviewed again or removed.In line with the CVR results, the CVI values also indicate a very good validity of the tool. 40,41Moreover, the reliability of the knowledge assessment questionnaire, assessed using the Kuder-Richardson coefficient, was found to be 0. Bausell's model. 40e reliability of performance questionnaires with 80 items in four assessment tools was confirmed using Cronbach's ⍺ values: Poster and pamphlet (0.880), Powerpoint (0.871), and role-playing (0.890).Reliability coefficients typically range from 0 to 1, with values above 0.90 indicating excellent reliability, values between 0.80 and 0.90 considered very good for classroom tests, and values between 0.70 and 0.80 considered good for classroom tests. 44,45Based on research findings, all fundamental values for performance measurement tools have been reported to be over 0.870, indicating that the designed tools exhibit excellent reliability.
7][48]  knowledge and the reliability coefficient of knowledge was 0.92. 49 the study of Rezaei et al., the CVI was equal to 0.922 and Cronbach's ⍺ coefficient was equal to 0.784, which was consistent with the results of our study. 50In the study of Marashi et al.
regarding the design of the health faculty students' awareness questionnaire in the field of AIDS, the reliability of the tool was 0.79, which is in line with this study. 51In a research conducted by Nasiri et al. 52   the course satisfaction questionnaire had a high CVI (0.949) and CVR (0.861) and a high overall reliability of 0.928.In addition to the total reliability, the reliability of each of the components has also been evaluated at a favorable level (0.66 to 0.830).It is worth mentioning that the reliability is greatly affected by the sample size.Although the number of samples was 50 people, the reliability value of the components is acceptable and suitable.
In assessing questionnaire reliability, it is important to consider the impact of individual items on the overall reliability of the scale.The "If item deleted" analysis provides insights into how removing specific items affects the reliability of the questionnaire.By analyzing statistics such as average scale values, variance, correlation coefficients, and Cronbach's ⍺ after deleting an item, researchers can determine if removing certain items would enhance the overall reliability of the questionnaire.
Obviously, if by removing an item, the reliability value of the items decreases, it indicates that that item had a positive effect on the reliability.
As in all research tools, including satisfaction measurement, it shows that the impact of all items on total reliability was positive. 55Various tools have been designed to measure students' satisfaction with the educational environment.The research of Madvari 56 was conducted in relation to the validity and reliability of the employee satisfaction questionnaire, the questionnaire designed for employee satisfaction was higher than the standard in terms of CVI (0.79) and CVR (0.55).The ⍺ coefficient obtained showed the high internal consistency of the questionnaire, also in the study of Torkzadeh et al. 35 the academic satisfaction scale of students had high validity and reliability.In the study T A B L E 4 Psychometric characteristics of each item in the performance assessment for Pamphlet tool.It is important to note that the findings of this research provide a set of common primary tools that are typically taught in technology education courses, but due to the limited sample size, it was not possible to establish construct validity, and it is necessary to conduct a comprehensive exploratory and confirmatory analysis to ensure the composition of the tools in in forthcoming studies.
experts specializing in medical education (3), health education (3), and educational technology (4) were consulted with purposive sampling method.The criteria for expert selection included having a relevant academic background and a minimum of 3 years of teaching experience in the educational technology course.Individuals willing to participate were included in the research, with the option to withdraw if they chose not to continue collaborating.For the quantitative section and testing of the tools, Undergraduate students majoring in public health from Asadabad medical college who had taken the educational technology course in 2022 were selected.A practical session of the educational technology course was conducted with a total of 50 participants selected.The criteria for student selection included enrollment in the educational technology course, no prior completion of this course, or participation in related open courses on educational technology.Similar to the expert selection process, students willing to engage in the research could opt-out if they decided not to continue participation.
experts' opinion, an instrument consisting of 80 questions in four checklists (20 questions in each checklists) including; Posters, Pamphlets, Role plays, and slides prepared in PowerPoint software were designed.Subscales of poster and pamphlet items included four components: Configuration, Perspective, Content, and Visual elements.The subscales of role-playing checklists included the three components: Role mastery, Design and Implementation, and control of conditions.The subscales of PowerPoint checklists included five components: Configuration, arrangement of content, scientific content, adjusted content, and Emphasis and attention were compiled.The answers to the performance questions were measured based on the scale of yes with 1 score, components included: Motivation (2 Items), Learning (3 Items), Problem solving (3 Items), Team working (3 Items), Interaction (3 Items), Time management (2 Items), and Application (2 Items) was measured with a 5point Likert scoring scale (very high, high, medium, low and very low) with a score range of 1-5.The minimum score was 1 and the maximum score was 5, and was acceptable cut-off-point score = 3 (Table

6 |
Data collection and data analysis Written ethical approval was taken from the SMART University of Medical Sciences' local ethics committee (approval number (Ref.No.IR.VUMS.REC.1401.017))and written informed consent obtained from all the participants.
Ahanchian54 in relation to the validation of the performance evaluation questionnaire is consistent with the results of the present study.Satisfaction: Based on the results of the present study, in the satisfaction section, 18 questions in seven areas with a 5-option Likert scale were designed in the field of motivation, learning, problem solving, team activity, interaction, time, and application, which actually express the satisfaction of students with the quality of the course.It is practical.Based on the findings of the present study, Psychometric characteristics of each item in the knowledge assessment tools.
Kuder-Richardson coefficient, specifically the Kuder-Richardson Formula 20 (KR-20), is a measure of internal consistency reliability used for tests with dichotomous choices, where each question has two possible answers (right or wrong).This coefficient assesses how consistent the results from the test are and how well the test measures what it is intended to measure.The KR-20 formula calculates the reliability of a test by considering the total number of pamphlets, and role-play tools, the CVR was examined.The CVR values for the checklist of posters (0.906), pamphlets (0.877), PowerPoint (0.960), and role-play tools (0.980) were confirmed.With 10 experts involved, a minimum agreement score of 0.62 was expected.The findings revealed excellent content validity scores for all tools based on the CVR. 39,41Moreover, the CVI scores exceeding T A B L E 2 Psychometric characteristics of each item in the performance assessment for Poster tool.
53ed.Based on the results, CVR, CVI, and Cronbach's ⍺ coefficient are 0.54, 0.728, and 0.782, respectively, which CVR, CVI, and Cronbach's ⍺ coefficient were higher in our research.In the study of Mohammad Abadi et al.53regarding the design and validation of the performance questionnaire of knowledge-based companies, with 17 statements in two dimensions, the reliability criterion of the entire T A B L E 3 The validity of the tool in this study was assessed through face and content validity.It is anticipated that the construct validity T A B L E 5 Psychometric characteristics of each item in the performance assessment for PowerPoint tool.Psychometric characteristics of each item in the performance assessment for Role Play tool.Psychometric characteristics of each item in the satisfaction assessment tool. of educational content production has expanded, and there is a need to develop tools for assessing the quality of this content.However, since new technologies are constantly evolving, and with the emergence of artificial intelligence and machine learning, new types of educational content and materials are gaining attention, these tools and indicators must undergo continuous updates.
5 | CONCLUSIONSPractical and experiential learning is less structured and more reliant on opportunities compared with purely theoretical education.As a result, it may sometimes receive less attention or not be evaluated with concrete measures.For instance, the course on educational technology in the achieve educational goals and are particularly suitable for various types of educational content, incorporating the perspectives of educational experts, health professionals, and experienced tool developers.Additionally, the tool's credibility has been confirmed by participants' feedback during the course (students), making it potentially applicable in similar fields.In particular, with the emergence of new technologies, the T A B L E 6 T A B L E 7 field